A Survey of Quality Assurance Practices in Biomedical Open Source Software Projects

Background Open source (OS) software is continuously gaining recognition and use in the biomedical domain, for example, in health informatics and bioinformatics. Objectives Given the mission critical nature of applications in this domain and their potential impact on patient safety, it is important to understand to what degree and how effectively biomedical OS developers perform standard quality assurance (QA) activities such as peer reviews and testing. This would allow the users of biomedical OS software to better understand the quality risks, if any, and the developers to identify process improvement opportunities to produce higher quality software. Methods A survey of developers working on biomedical OS projects was conducted to examine the QA activities that are performed. We took a descriptive approach to summarize the implementation of QA activities and then examined some of the factors that may be related to the implementation of such practices. Results Our descriptive results show that 63% (95% CI, 54-72) of projects did not include peer reviews in their development process, while 82% (95% CI, 75-89) did include testing. Approximately 74% (95% CI, 67-81) of developers did not have a background in computing, 80% (95% CI, 74-87) were paid for their contributions to the project, and 52% (95% CI, 43-60) had PhDs. A multivariate logistic regression model to predict the implementation of peer reviews was not significant (likelihood ratio test = 16.86, 9 df, P = .051) and neither was a model to predict the implementation of testing (likelihood ratio test = 3.34, 9 df, P = .95). Conclusions Less attention is paid to peer review than testing. However, the former is a complementary, and necessary, QA practice rather than an alternative. Therefore, one can argue that there are quality risks, at least at this point in time, in transitioning biomedical OS software into any critical settings that may have operational, financial, or safety implications. Developers of biomedical OS applications should invest more effort in implementing systemic peer review practices throughout the development and maintenance processes.


Data Protection
The online survey tool was installed on a server computer located at the researchers' lab at UMBC. This tool provides password-controlled access to its administrators, the researchers, to enter and view respondents' names and email addresses.

Development and testing
The survey was developed and tested using the OS tool, PhpSurveyor. The researchers tested this tool several times and assured its functionality and usability before the actual survey took place.
Open vs. Closed The survey was a closed survey. The survey tool automatically created e-tokens (long and complex URLs) that allowed access to the on-line survey form, and emailed them to the respondents. Each potential respondent received one token.

Contact Mode
Respondents received e-mails explaining the goals and purposes of the survey and asking their contribution. Emails included the tokens which took the potential respondents to the on-line survey form. After the first email, reminder e-mails were sent over a period of six weeks for non-respondents after the first week.

Recruitment process and the description of the sample having access to the questionnaire
Advertising No advertising was made.
Web / E-mail The contacts were made by e-mails. However, the survey was web-based. The respondents used their web-browsers to respond. The data was collected automatically after their submission on researchers' computer that hosts the web server and MySql database. All data kept in this database is password protected.

Context
Following the special URL (token) given in the e-mail, the respondents were only able to view the survey form. They were not shown any other content.
Mandatory/ Voluntary The respondents were able to view the survey form without filling out the survey and submitting their answers.
Responding to the survey was voluntary. Upon clicking on the submission button, it was checked whether the response was a complete response or not. The respondent was reminded and asked to answer in case any question was left unanswered.

Incentives
No incentive was given other than telling respondents that they would be informed about any resulting report or publication of this research.

Time/Date
The survey was conducted between Oct 10 and Nov 17, 2005.

Randomization
No items or questionnaires were randomized.
Adaptive Questioning Adaptive or conditional questioning was not used.

Number of Items
The survey questions relevant for this paper are shown in Appendix 1.

Number of screens
The whole questionnaire was a single page, the respondents replied by scrolling down to the next question.
Completeness check Each submitted response was checked for completeness. This functionality was available in the survey instruments by making all of the questions mandatory.

Review Step
The respondents could review their answers before submission by scrolling up the page.

Individual Response Rate
The individual response rate was 18 Log file analysis Some e-mail addresses were not valid anymore. The emails sent to these addresses were returned, and they were detected from the e-mail logs of the root account of our server machine. We excluded these individuals in calculating our response rate.

Registration
The user could view the survey page only until s/he submitted the completed survey. The survey was never shown again to this user with the token that he used.

Handling of incomplete questionnaires
All of the survey forms were completed since the instrument checked for completeness and only accepted the complete forms.

Analysis
Questionnaires submitted with atypical time stamp Time to fill out the survey was not tracked. However, respondents only had one opportunity to submit the survey with their e-token, after which that token was disabled.